Calculating global importance of documents based on global hitting times

ABSTRACT

A calculate importance system calculates the global importance of a web page based on a “mean hitting time.” Hitting time of a target web page is a measure of the minimum number of transitions needed to land on the target web page. Mean hitting time of a target web page is an average number of such transitions for all possible starting web pages. The calculate importance system calculates a global importance score for a web page based on the reciprocal of a mean hitting time. A search engine may rank web pages of a search result based on a combination of relevance of the web pages to the search request and global importance of the web pages based on a global hitting time.

BACKGROUND

Many search engine services, such as Google and Yahoo, provide forsearching for information that is accessible via the Internet. Thesesearch engine services allow users to search for display pages, such asweb pages, that may be of interest to users. After a user submits asearch request (i.e., a query) that includes search terms, the searchengine service identifies web pages that may be related to those searchterms. To quickly identify related web pages, the search engine servicesmay maintain a mapping of keywords to web pages. This mapping may begenerated by “crawling” the web (i.e., the World Wide Web) to identifythe keywords of each web page. To crawl the web, a search engine servicemay use a list of root web pages to identify all web pages that areaccessible through those root web pages. The keywords of any particularweb page can be identified using various well-known informationretrieval techniques, such as identifying the words of a headline, thewords supplied in the metadata of the web page, the words that arehighlighted, and so on. The search engine service identifies web pagesthat may be related to the search request based on how well the keywordsof a web page match the words of the query. The search engine servicethen displays to the user links to the identified web pages in an orderthat is based on a ranking that may be determined by their relevance tothe query, popularity, importance, and/or some other measure.

One well-known technique for page ranking is PageRank, which is based onthe principle that web pages will have links to (i.e., “out links”)important web pages. The importance of a web page is based on the numberand importance of other web pages that link to that web page (i.e., “inlinks”). PageRank is based on a random surfer model of visiting webpages of a web graph (vertices representing web pages and linksrepresenting hyperlinks) and represents the importance of a web page asthe stationary probability of visiting that web page. In the randomsurfer model, a surfer visiting a current page will visit a next page byrandomly selecting a link of the current web page. If the current webpage has three out links to target web pages, then the probability ofvisiting each target web page from the current web page is ⅓. PageRankis thus based on a Markov random walk that only depends on theinformation (e.g., hyperlink) of the current web page.

A web graph may be represented as G=<V,E>, where V={1, 2, . . . , n} isthe set of vertices and E={<i,j>|i,jεV} is the set of edges. The linksbetween web pages can be represented by an adjacency matrix A, whereA_(ij) is set to one when there is an out link from a source web page ito a target web page j. The importance score w_(j) for web page j can berepresented by the following:

w_(j)=Σ_(i)A_(ij)w_(i)  (1)

This equation can be solved by iterative calculations based on thefollowing:

A^(T)w=w  (2)

where w is the vector of importance scores for the web pages and is theprincipal eigenvector of A^(T).

PageRank may also factor in that a surfer may randomly select a web pageto visit next that is not linked to by the current web page. Thus, thesurfer may next visit a target web page of the current web page with aprobability of α and next visit a randomly selected web page with aprobability of 1−α. To factor in this random selection of web pages,PageRank generates an initial transition matrix P by normalizing eachnon-zero row of the adjacency matrix with the sum of its elements.PageRank then sets each element of a zero row in matrix P to 1/n togenerate transition probability matrix P. The model of representing therandom selection of links of target web pages and the random selectionof web pages can be represented by the following:

P=α P +(1−α)U  (3)

where P is the combined transition probability matrix and U is a uniformprobability distribution matrix in which each element is set to 1/n.PageRank considers the stationary distribution π(π₁, π₂, . . . ,π_(n))^(T) of the transition probability matrix P to represent theimportance of each web page. PageRank may compute the stationarydistribution through an iterative process as represented by thefollowing:

π(t+1)=( P )^(T)π(t)  (4)

where π(0)=(1, 1, . . . , 1)_(n) ^(T), t represents the iteration count,and the iterative process continues until π converges on a solution.

Although PageRank can be very useful, in part because it is aquery-independent measure of importance, it is especially susceptible to“link spamming.” “Spamming” in general refers to a deliberate actiontaken to unjustifiably increase the rank, relevance, popularity,importance, and so on of a web page or a web site. In the case of linkspamming, a spammer can manipulate links to unjustifiably increase theimportance of a web page. For example, a spammer may provide a web pageof useful information with hidden links to spam web pages. When many webpages point to the useful information, the importance of the spam webpages is indirectly increased. As another example, many web sites, suchas blogs and web directories, allow visitors to post links. Spammers canpost links to their spam web pages to directly or indirectly increasethe importance of the spam web pages. As another example, a group ofspammers may set up a link exchange mechanism in which their web sitespoint to each other to increase the importance of the web pages of thespammers' web sites.

Web spam presents problems for various techniques that rely on web data.For example, a search engine service that orders search results in partbased on relevance, popularity, or importance of web pages may rank spamweb pages unjustifiably high because of the spamming. As anotherexample, a web crawler may spend valuable time crawling the links ofspam web sites, which increases the overall cost of web crawling and mayreduce its effectiveness.

PageRank is especially susceptible to link spamming because it is basedon the concept of a “returning time” of a web page. Returning time of aweb page is a measure of the number of transitions needed to return tothe web page starting from the web page itself. The importance score ofa web page is the reciprocal of the returning time. Thus, when a webpage has a small returning time, it will have a large importance score.Returning time may represented by the following:

T _(i) ⁺=min{t≧1:X _(t) =i}  (5)

where T_(i) ⁺ represents the returning time for web page i and X_(t)represents a discrete-time Markov chain of the transition probabilitymatrix for time t. The stationary transition probability, and thusimportance, can be represented by the following:

π_(i)=1/E _(i) T _(i) ⁺  (6)

where represents the stationary probability for web page i and 47;represents the expected returning time. Since the Markov chain startsfrom a target page i itself, the behavior of the random walk is largelyaffected by the local structure around page i. Link spammers can createan arbitrary local structure around a web page to take advantage of thisdrawback of PageRank. A link spammer can set up a star-structured linkfarm in which a central web page contains links to many other boostingweb pages and those boosting web pages contain links only to the centralweb page. In such a case, the random walk from the central web page istrapped in this star-like local structure, and the mean returning timecan be significantly reduced, and therefore importance is increased. Inthe random walk model without a possibility of jumping to a randomnon-linked-to web page (α=1), all possible series of random walktransitions will be between the central web page and one of its boostingweb pages. The mean returning time is only 2. Even when the possibilityof transitioning to a random web page is factored in (α<1), such a linkfarm can significantly reduce the mean returning time and increaseimportance.

SUMMARY

A method and system for calculating global importance of documents basedon a global hitting time is provided. A calculate importance systemcalculates the global importance of a web page based on a “mean hittingtime.” Hitting time of a target web page is a measure of the minimumnumber of transitions needed to land on the target web page. Meanhitting time of a target web page is an average number of suchtransitions for all possible starting web pages. A hitting time that isbased on a hitting time with a starting page other than the target webpage is considered to be a global hitting time. The calculate importancesystem calculates a global importance score for a web page based on thereciprocal of a global hitting time. The hitting times of the web pagesof link spam are likely to be overwhelmed by the hitting times of thehundreds of millions of other web pages. A search engine may rank webpages of a search result based on a combination of relevance of the webpages to the search request and global importance of the web pages basedon a global hitting time.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram that illustrates components of the calculateimportance system in one embodiment.

FIG. 2 is a flow diagram that illustrates the overall processing of thecalculate global importance component of the calculate importance systemin one embodiment.

FIG. 3 is a flow diagram that illustrates detailed processing of thecalculate global importance component of the calculate importance systemin one embodiment.

FIG. 4 is a flow diagram that illustrates the processing of thecalculate power diagonals component of the calculate importance systemin one embodiment.

FIG. 5 is a flow diagram that illustrates the processing of the searchengine component of the calculate importance system in one embodiment.

FIG. 6 is a flow diagram that illustrates the processing of the rank webpages component of the calculate importance system in one embodiment.

DETAILED DESCRIPTION

A method and system for calculating global importance of documents basedon a global hitting time is provided. In some embodiments, a calculateimportance system calculates the global importance of a web page basedon a “mean hitting time.” Hitting time of a target web page is a measureof the minimum number of transitions needed to land on the target webpage. The hitting time depends on the starting web page. For example,the hitting time from a starting web page that has a direct link to thetarget web page is 1, whereas the hitting time from a starting web pagethat is indirectly linked to the target web page via a sequence of fourother web pages is 5. Mean hitting time of a target web page is anaverage number of such transitions for all possible starting web pages.Hitting time is different from returning time in that returning time isa special case of hitting time in which the starting web page and thetarget web page are the same web page. A hitting time that is based on ahitting time with a starting page other than the target web page isconsidered to be a global hitting time. Mean hitting time is one exampleof a global hitting time.

In one embodiment, the calculate importance system calculates a globalimportance score for a web page based on the reciprocal of a meanhitting time. Because the mean hitting time may be an average based onthe hitting time of all (or at least very many) web pages, thecontribution of link spam, such as a star-structured link farm, to theglobal importance of a web page is much less than for an importancebased on returning time. The hitting times of the web pages of link spamare likely to be overwhelmed by the hitting times of the hundreds ofmillions of other web pages. A search engine may rank web pages of asearch result based on a combination of relevance of the web pages tothe search request and global importance of the web pages based on aglobal hitting time. In this way, the contribution of link spam toimportance of a web page can be greatly reduced.

The calculate importance system may represent hitting time as follows:

T_(i)=min{t≧0:X_(t)=i}  (7)

where T_(i) represents the hitting time of web page i and X_(t)represents a discrete-time Markov chain of the transition probabilitymatrix for time t. The calculate importance system represents the globalimportance of a web page as the reciprocal of the mean hitting timestarting from any distribution over the whole web graph as representedby the following:

GR _(μ)(i)=1/E _(μ) T _(i)  (8)

where μ represents any chosen global distribution, that is, theprobability of each web page being the starting web page. The meanhitting time is represented as follows:

E_(μ)T_(i)=Σ_(j)μ_(j)E_(j)T_(i)  (9)

where μ_(j) represents the probability that a random surfer chooses webpage j as the starting web page and E_(j)T_(i) represents the expectedhitting time with starting web page j and target web page i. By takingthe mean of the expected hitting time starting from all web pages, thelocal structure surrounding a target web page, including link spam, isgreatly attenuated. Although link spammers could change the localstructure of a target web page to decrease E_(j)T_(i) with web pagesaround the target web page i, those surrounding web pages under the linkspammers' control are only an extremely small portion of the overall webgraph. The contributions of link spammers' efforts are attenuated byaveraging over expected hitting times from starting web pages not undertheir control. Web pages that gain high importance based on low globalhitting times are globally important web pages, such as the main webpages of major search engines, large companies, popular onlinecommunities, and so on. These web pages have a central role in the wholeweb graph and are accessible from most web pages within a fewtransitions.

In some embodiments, the calculate importance system may use differentinitial distributions μ depending on the objectives of calculating theimportance. The calculate importance system may use any one orcombination of the following distributions:

-   -   1) a stationary probability distribution,    -   2) a uniform distribution over all web pages,    -   3) a distribution over labeled non-spam pages or other known        desirable web pages, and    -   4) a user-specific distribution (e.g., web pages of a user's        favorite web sites).

In some embodiments, the importance calculation system calculates theglobal importance of web pages using a global hitting time and astationary probability distribution as the initial distribution,referred to a stationary global importance. The calculate importancesystem calculates the importance by converting the expectation of arandom variable to an expression using the diagonal of t-ordertransition matrix P_(t). The calculate importance system calculates theimportance using the fundamental matrix as represented by the following:

Z≡(I−P+eπ^(t))⁻¹  (10)

where I is the identity matrix. (See Aldous, D. and Fill, J.,“Reversible Markov Chains,” Monograph, 1994.) The relationship betweenthe mean hitting time given the initial stationary probabilitydistribution and Z is represented by the following:

E _(π) T _(i) =Z _(ii)/π_(i)

E _(j) T _(i)=(Z _(ii) −Z _(ij)/π_(i)  (11)

where Z_(ij) is represented by the following:

Z _(ij)=δ_(ij)+Σ_(t=1) ^(∞)(P _(ij) ^(t)−π_(j))  (12)

where δ_(ij) is equal to 1 when i equals j and 0 otherwise. Thecalculate importance system can approximate Z_(ij) using the first mterms of Equation 12. Thus, the approximation of Z_(ij) can berepresented by the following:

{circumflex over (Z)} _(ij)=δ_(ij)+Σ_(t=1) ^(m)(P _(ij)^(t)−π_(j))  (13)

where {circumflex over (Z)}_(ij) represents the approximation of Z_(ij).The value of m may be, for example, 5.

FIG. 1 is a block diagram that illustrates components of the calculateimportance system in one embodiment. The calculate importance system 110is connected to web site servers 150 and user computing devices 160 viacommunications link 140. The calculate importance system may include acrawler component 111, a keyword/web page index 112, an adjacency matrixstore 113, and an importance score store 114. The crawler component maycrawl the web pages of the web sites and store a mapping of keywords toweb pages that contain the keywords in the keyword/web page index. Thecrawler component may also store a representation of the adjacencymatrix in the adjacency matrix store.

The calculate importance system may also include a calculate globalimportance component 121, a calculate initial distribution component122, and a calculate transition diagonals component 123. The calculateglobal importance component calculates the global importance of each webpage. The calculate global importance component invokes the calculateinitial distribution component to calculate the initial distribution.The calculate global importance component then calculates globalimportance of the web page according to Equation 10. The calculateglobal importance component invokes the calculate transition diagonalscomponent to generate the values for Z_(ij) of Equation 11. Thecalculate global importance component stores the global importance ofeach web page as an importance score in the importance score store.

The calculate importance system may include a search engine 131 and arank web pages component 132. The search engine receives search requestsand uses the keyword/web page index to identify matching web pages as asearch result. The search engine then invokes the rank web pagescomponent to rank the web pages of the search result based on relevanceto the search request and the global importance of the web pages asindicated by the importance score store.

The computing device on which the calculate importance system isimplemented may include a central processing unit, memory, input devices(e.g., keyboard and pointing devices), output devices (e.g., displaydevices), and storage devices (e.g., disk drives). The memory andstorage devices are computer-readable media that may be encoded withcomputer-executable instructions that implement the calculate importancesystem, which means a computer-readable medium that contains theinstructions. In addition, the instructions, data structures, andmessage structures may be stored or transmitted via a data transmissionmedium, such as a signal on a communication link. Various communicationlinks may be used, such as the Internet, a local area network, a widearea network, a point-to-point dial-up connection, a cell phone network,and so on.

Embodiments of the system may be implemented in and used by variousoperating environments that include personal computers, servercomputers, hand-held or laptop devices, multiprocessor systems,microprocessor-based systems, programmable consumer electronics, digitalcameras, network PCs, minicomputers, mainframe computers, computingenvironments that include any of the above systems or devices, and soon.

The calculate importance system may be described in the general contextof computer-executable instructions, such as program modules, executedby one or more computers or other devices. Generally, program modulesinclude routines, programs, objects, components, data structures, and soon that perform particular tasks or implement particular abstract datatypes. Typically, the functionality of the program modules may becombined or distributed as desired in various embodiments. For example,a separate computing system may crawl the web and generate the adjacencymatrix and the keyword/web page mapping. Also, the search engine may behosted on a separate computing system.

FIG. 2 is a flow diagram that illustrates the overall processing of thecalculate global importance component of the calculate importance systemin one embodiment. In block 201, the component generates an initialdistribution for the web pages whose importance is to be calculated. Inblock 202, the component generates the mean hitting times for each ofthe web pages based on the initial distribution. In blocks 203-205, thecomponent loops setting the global importance for each web page. Inblock 203, the component selects the next web page. In decision block204, if all the web pages have already been selected, then the componentcompletes, else the component continues at block 205. In block 205, thecomponent sets the importance of the selected web page based on theinverse of the mean hitting time for the web page and then continues atblock 203.

FIG. 3 is a flow diagram that illustrates detailed processing of thecalculate global importance component of the calculate importance systemin one embodiment. The component is passed an indication of the numberof transitions to use in the approximation. The component approximatesthe global importance based on Equations 11 and 13. In block 301, thecomponent generates an initial probability distribution for the webpages. In some embodiments, the component uses the stationaryprobabilities derived from a probability transition matrix based on theadjacency matrix as represented by Equation 4. In block 302, thecomponent invokes a calculate power diagonals component to calculate thediagonals P_(ii) ^(m) used in Equations 11 and 13. In blocks 303-306,the component loops approximating the global importance of each webpage. In block 303, the component selects the next web page. In decisionblock 304, if all the web pages have already been selected, then thecomponent returns the global importance scores, else the componentcontinues at block 305. In block 305, the component calculates the valueof Equation 13 for the selected web page using the diagonal value forthe selected web page calculated in block 302. In block 306, thecomponent sets the global importance score for the selected web page tothe stationary distribution of the selected web page divided by thevalue calculated in block 305. The component then loops to block 303 toselect the next web page.

FIG. 4 is a flow diagram that illustrates the processing of thecalculate power diagonals component of the calculate importance systemin one embodiment. The component calculates the diagonal values for thepowers of the transition probability matrix by calculating the powers ofthe transition probability matrices—a brute-force approach. Thecomponent is passed an indication of the number of transitions. In block401, the component initializes the zero power of the transitionprobability matrix to the initial transition probability matrix. Inblock 402, the component increments an indication of the current power.In decision block 403, if the current power is greater than the numberof transitions, then the component returns, else the component continuesat block 404. In block 404, the component calculates the next power ofthe transition probability matrix and then loops to block 402 to selectthe next power. Alternatively, the calculate importance system may use arandom sampling approach to compute the diagonals of the powertransition probability matrix as described in Kamvar, S., Haveliwala,T., Manning, C., and Golub, G., “Exploiting the Block Structure of theWeb for Computing PageRank,” Stanford University Technical Report, 2003.

FIG. 5 is a flow diagram that illustrates the processing of the searchengine component of the calculate importance system in one embodiment.In block 501, the component inputs a query from a user. In block 502,the component finds web pages that match the query using the keyword/webpage index. In block 503, the component invokes the rank web pagescomponent to rank the matching web pages. In block 504, the componentdisplays the web pages in ranked order. The component then completes.

FIG. 6 is a flow diagram that illustrates the processing of the rank webpages component of the calculate importance system in one embodiment.The component is passed web pages and ranks the web pages based onrelevance and global importance. In block 601, the component retrievesthe importance scores of the web pages. In block 602, the componentselects the next web page. In decision block 603, if all the web pageshave already been selected, then the component continues at block 606,else the component continues at block 604. In block 604, the componentcalculates the relevance of the web page to the query. In block 605, thecomponent generates a rank score for the selected web page by combiningthe global importance of the web page with the relevance of the webpage. The component then loops to block 602 to select next web page. Inblock 606, the component sorts the web pages based on the rank scoresand returns the ranked web pages.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims. One skilled in the art willappreciate that a document can include any information content thatcontains links or otherwise identifies other content. For example, adocument may be a web page with links to other web pages, a scholarlyarticle with citations to other scholarly articles, a judicial opinionwith citations to other judicial opinions, a patent with citations toother patents, and so on. Accordingly, the invention is not limitedexcept as by the appended claims.

1.-20. (canceled)
 21. A method in a computing device for generating animportance score for documents having links between the documents, themethod comprising: providing an initial probability distribution foreach document; providing a transition probabilities indicating theprobability of transitioning between documents; and for each document,calculating by the computing device a probability of starting from andreturning to that document after a number of transitions based on theinitial probability distribution and the transition probabilities; andsetting the importance score of the document based on the calculatedprobability.
 22. The method of claim 21 wherein the calculating of theprobability includes multiplying the transition probabilities bythemselves the number of times.
 23. The method of claim 21 wherein thecalculating of the probability includes using a random samplingtechnique.
 24. The method of claim 21 wherein the documents are webpages and the web pages are ranked at least in part based on theimportance scores.
 25. The method of claim 24 wherein the web pages aresearch results of a search request and the web pages are ranked at leastin part based on relevance of the web pages to the search request. 26.The method of claim 21 wherein the initial probability distribution isbased on a stationary distribution of the transition probabilities. 27.The method of claim 21 wherein the initial probability distribution isuniform.
 28. The method of claim 21 wherein the initial probabilitydistribution is zero for documents considered to be spam.
 29. The methodof claim 21 wherein the initial probability distribution is personalizedto a user.
 30. The method of claim 21 wherein the transitionprobabilities represent the probabilities of using the links totransition between documents.
 31. The method of claim 21 wherein thetransition probabilities represents the probability of using the linksto transition between documents.
 32. A computer-readable storage deviceencoded with instructions for controlling a computing device to rank webpages, by a method comprising: generating importance scores for webpages that are based on a mean hitting time for the web pages, the meanhitting time for a target web page being based on an average of hittingtimes of transitioning from starting web pages to the target web page;and ranking the web pages based at least in part on the importancescores of the web pages.
 33. The computer-readable storage device ofclaim 32 wherein the mean hitting time is based on an initialprobability distribution of the starting web pages that is derived froma stationary probability of transition probabilities of the web pages.34. The computer-readable storage device of claim 32 wherein the meanhitting time is based on an initial probability distribution of startingpages that is user-specific.
 35. The computer-readable storage device ofclaim 32 wherein web pages that are ranked are search results of asearch request.
 36. The computer-readable storage device of claim 32wherein the web pages are further ranked based on relevance of a webpage to the search request.
 37. A computing system for rankingdocuments, comprising: an importance store having importance scores fordocuments, the importance score for a document based on a global hittingtime, the global hitting time for a target document being based onnumber of transitions from a starting document to land on the targetdocument; a memory storing computer-executable instructions of: acomponent that identifies documents of a search result for a searchrequest; a component that determines relevance of each identifieddocument to the search request; and a component that ranks theidentified documents based at least in part on the importance score ofthe documents and the relevance of the documents to the search request;and a processor that executes the computer-executable instructionsstored in the memory.
 38. The computing device of claim 37 wherein theglobal hitting time is based on an initial probability distribution ofthe starting documents that is derived from a stationary probability oftransition probabilities of the documents.
 39. The computing device ofclaim 38 wherein the global hitting time is a mean hitting time.
 40. Thecomputing device of claim 38 wherein the documents are web pages and atransition includes following a link from one web page to another webpage.